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METHOD FOR DISPLAY TIME STAMPING 
AND SYNCHRONIZATION OF MULTIPLE 
VIDEO OBJECT PLANES 

TECHNICAL HELD 

This invention is useful in the coding of digital audio 
visual materials where there is a need for synchronizing 
more than one independently encoded audio visual objects 
for presentation. It is especially suitable when the temporal 
sampling of the audio visual material is not the same. 

BACKGROUND ART 

In MPEG-1 and MPEG-2 the input video consists of 
image frames sampled at regular time intervals. This repre- 
sents the finest temporal resolution of the input FIG. 1 shows 
a video sequence with fixed frame rates where the image 
frames are sampled at regular intervals. In the coded repre- 
sentation of the video sequence using the MPEG-1 and 
MPEG-2 standards the display order of the decoded frames 
is indicated by the temporal reference. This parameter 
appears at the picture header of the bitstream syntax. The 
value of this parameter is incremented by one for each 
decoded frame when examined in the display order. 

In the H.263 standard it is possible to skip frames and thus 
decode a variable frame rate video sequence. However the 
sampling of frames is still fixed. Thus the temporal reference 
method used in MPEG-1 and MPEG-2 is still appropriate 
and only need to be modified such that the increment is not 
in steps of 1 but in steps of 1 plus the number of non- 
transmitted pictures at the input frame rate. 

Currently work is being done in the area of coding video 
as separate objects in multiple video object planes. This 
represents a new dimension in the decoding and synchro- 
nization of the respective video objects. It is expected that 
these different video object planes may come from several 
sources and can have very different frame rates. Some of the 
objects may be rendered and have almost continuous tem- 
poral sampling rate. These video objects planes are com- 
bined into a composite image for display. Therefore some 
kind of synchronization is required for the composition. It is 
possible that the display frame rate is different from any of 
the frame rates of the video object planes. FIG. 2 shows an 
example of two video object planes having different and 
varying frame rates compared to one another. Even if a 
common frame rate can be found between the two video 
object plane, it is not automatic that this frame rate will be 
the same as the output frame rate of the compositor. 

Hereafter we will refer to the problem in the video 
domain. However the same principles of this invention can 
be extended to the audio domain as well as combinations of 
the two. 

From the above it is clear that the current state of the art 
does not cater for the synchronization of video object planes. 
The state of the art also does not provide a conomon temporal 
reference when the different video object planes have dif- 
ferent frame rates which are not multiples of one another. 

The first problem is how to provide a common local tune 
base mechanism for each video object plane. This time base 
will have to be able to provide very fine temporal granularity 
and at the same time be able to cope with possibility of very 
long intervals between two successive instances of the video 
object planes. 

The second problem is how to provide a mechanism for 
synchronizing video object planes having different frame 
rates. 
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DISCLOSURE OF INVENTION 

The above problems can be solved by implementing a 
common temporal resolution to be used for ail the local time 
base. In order to cater for the wide range of temporal 
^ granularity the local time base is partitioned into two dif- 
ferent parts. The first part containing the fine granularity 
temporal resolution which provide a short time base. The 
second part contain a coarse granularity temporal resolution 
which provide a long time base. The short time base is 
included at each the video object planes to provide temporal 
reference to the instances of the video objects. The short 
time base is then synchronized to the longtime base which 
is common to all the video object planes. It is used to 
synchronize aM the different video object planes to a com- 
mon time base provided by the master clock. 

According to a fist aspect of the present invention, a 
method of encoding a local time base of an audio visual 
sequence m ±e compressed data comprises the steps of: 

obtaining instances of the audio visual sequence by tem- 
poral sampling; 

determining the local time base of the said instances to be 
encoded into compressed data; 

encoding the said local time base in two parts, comprising 
25 of a modulo time base that marks the occurrence of a set of 
time reference spaced evenly at a specific interval on the 
local time base and a time base increment relative to the said 
evenly spaced time reference; 

inserting the modulo time base into the compressed data 
30 whenever the specific interval have elapsed; and 

inserting the time base increment within the compressed 
data of the said instances of the audio visual sequence. 

According to a second aspect of the invention, a method 
of encoding a local time base of an audio visual sequence in 
35 the compressed data comprises the steps of: 

obtaining instances of the audio visual sequence by tem- 
poral sampling; 

determining the local time base of the said instances to be 
encoded into compressed data; 

encoding the said instances in one of two methods com- 
prismg of a first method of compression without reference to 
any future instances and a second method of compression 
which fe?ei?li7'the future reconstructed instance; 

encoding the said local time base in two parts, comprising 
of a modulo time base daat marks the occurrence of a set of 
tune reference spaced evenly at a specific interval on the 
local time base and a time base increment 

encoding the time base increment for the instance com- 
pressed using the first compression method as an absolute 
value relative to the said evenly spaced tune reference; 

encoding the time base increment for the instance com- 
pressed using the second compression method as a relative 
value to the local time base of the previously compressed 
2j instance using the said first method; 

inserting the modulo time base into the compressed data 
whenever the specific interval have elapsed; and 

inserting the time base increment within the compressed 
data of the said mstances of the audio visual sequence. 
60 According to a third aspect of the mvention, a method of 
the first or the second aspect of the invention, wherein a 
plurality of compressed bitstreams containing local time 
base information encoded therein are multiplexed, further 
comprises, for multiplexing, the steps of: 
65 synchronizing the local time base of the individual com- 
pressed bitstream to a common time base by encoding and 
inserting the time base offset in the multiplexed bitstream; 
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examining each of the compressed bitstream for the next 
compressed instance to be placed mto the multiplexed 
bitstream until ail the compressed bitstreams have arrived at 
the modulo time base; 

inserting a common modulo time base into the multiplex 
bitstream and skipping the modulo time base of the com- 
pressed bitstreams; and 

repeating the last two steps until aH compressed bit- 
streams are exhausted. 

According to a fourth aspect of the invention, a method of 
the first or the second aspect of die invention, wherein a 
plurality of compressed bitstreams containing local time 
base information encoded therein are multiplexed and 
de-multiplexed» farther comprises, for de-multiplexing, the 
steps of: 

decoding the time base offset of the individual com- 
pressed bitstream; 

examining the multiplexed bitstream for the next com- 
pressed instance and placing the said instance into the 
appropriate compressed bitstream until a modulo time base 
is encountered in the multiplexed bitstream; 

inserting a modulo time base to each of the compressed 
bitstreams; and 

repeating the last two steps until the multiplexed bitstream 
is exhausted. 

According to a fifth aspect of the invention, a method of 
decoding a local time base of an audio visual sequence from 
the time base of the compressed data encoded, according to 
the first aspect of the invention, comprises the steps of: 

initializing the reference time base taking into account the 
time base offset; 

mcrementing die reference time base by a specific interval 
for each modulo time base decoded; 

decoding the time base increment of the compressed 
instance; and 

determining the decoding time base of the said instance 
by adding the said decoded time base increment value to the 
reference time base. 

According to a sixth aspect of the invention, a method of 
decoding a local time base of an audio visual sequence from 
the time base of the compregse<ha:.ta encoded, according to 
the second aspea of the invention, comprises the steps of 

initializing the reference time base taking into account of 
the time base offset; 

incrementing the reference time base by a specific interval 
for each modulo time base decoded; 

decoding the time base increment of the compressed 
instance; and 

determining the time base increment to be one of two 
types, absolute or relative, depending on the first or second 
compression method used in the encoding of the instance, 
respectively; 

determining the decoding time base of the said instance 
by adding the said decoded time base increment value to the 
reference time base if the time base is of the first type; and 

determining the decoding time base of the said instance 
by adding the said decoded time base mcrement value to the 
decoding time base of the previous instance encoded using 
the first compression method, if die time base increment is 
of the second type. 

BRIEF DESCRJFnON OF DRAWINGS 
The present invention will become more fully understood 
from the detailed description given below and tht accom- 
panying diagrams wherein: 
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FIG. 1 illustrates the temporal sampling of the prior art 
where the frames of the video sequence is sampled at regular 
intervals. 

FIG, 2 illustrates the concept of video object planes and 
5 the relationship with one another. The sampling of the video 
object planes can be irregular and the sampling period can 
vary drastically. 

FIG, 3A illustrates the current invention where the tem- 
poral reference of the video objects are mdicated by the 
modulo time base and the VOP time increment. Illustration 
uses I and P VOP only. 

FIG. 3B illustrates the current invention where the tem- 
poral reference of the video objects are mdicated by die 
modulo time base and thei VOP time increment. Illustration 
uses I, P and B VOR 

FIG, 4 illustrates an example of the ambiguity that can 
arise when the presentation order and the encoding order is 
different as a result of B-video object planes. 
20 FIG. 5 illustrates a solution to the ambiguity by employ- 
ing absolute and relative time bases. 

FIG. 6 illustrates the combining of two VOPs and their 
synchronization to the common time base by employing 
VOP time offsets. 

RG. 7 illustrates die flow diagram for the encodmg of the 
time base. 

FIG. 8 illustrates the flow diagram for the multiplexing of 
more than one video object planes. 
30 FIG, 9 illustrates the flow diagram for die de-multiplexing 
of more than one video object planes. 

FIG. 10 illustrates the flow diagram for the recovery of the 
presentation time stamp. 

FIG. 11 illustrates the block diagram for the operations of 
35 the bitstream encoder for encoding of the time base. 

FIG. 12 illustrates the block diagram for the operations of 
the bitstream decoder for decoding of the time base. 
FIG, 13 illustrates the time chart for forming a bitstream 

40 

BEST MODE FOR CARRYING OUT THE 
INVENTION 

This invention operates by providing two forms of syn- 
43 chromzation. The first is a short time base attached to the 
video object plane. Hereafter this time base will be referred 
to as the VOP time increment. The VOP time increment is 
the timing mechanism for the video object plane with 
respect to the long time base which is attached to a group of 
50 video object planes to be decoded and composed together. 
The long tune base is referred to as the modulo time base. 
The VOP thne increment and the modulo time based will 
tiien be used jointiy to determine the actual time base to use 
for the composition of the video object planes into the final 
55 composited sequence for display. 

In order to facilitate editing of tiie bitstream and combin- 
ing different video object planes from different sources into 
a new group of video object planes there is a need for a diird 
component to allow a fixed amount of offset between the 
60 local time base of the individual video object plane from the 
common time base. Hereafter this offset will be referred to 
as die VOP time offset This prevents the different video 
object planes from having to synchronize at a granularity 
eqmvalent to the modulo time base interval. This component 
65 should be kept constant for each video object plane within 
the group of video object planes that are multiplexed 
together. 
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First, the modulo time base is described. 

Moduio time base represent the coarse resolution of the 
local time base. It does not have a value like the VOP time 
increment. In fact it is more of a synchronization mechanism 
to synchronize the VOP time increment to the local time 
base of the video object plane. It is placed in the encoded 
bitstream as markers to indicate that the VOP time increment 
of the video object planes that follows are to be reset and that 
the reference to the local time base are to be incremented by 
one or more units of the modulo time base interval. In FIGS. 
3 A, 3B, 4, 5 and 6, the moduio time base is shown as a series 
of zero or more "1" terminated by a "0" inserted into the 
bitstream header before the VOP time increment. The num- 
ber of "'i" inserted into the bitstream depends on the number 
of units of modulo time base that has elapsed since the last 
code I or P-video object plane. At the encoder and decoder 
the modulo time base counter is incremented by one for each 
"1" encountered. Moduio time based counter is finite in 
length and therefore in a practical system the moduio time 
base is reset to zero when the maximum value is exhausted. 
In a typical video sequence, the video object planes form a 
group of VOP. Therefore the moduio time base is usually 
reset at the start of this group of VOP. 

Next, the VOP tmie increment is described. 

The VOP time increment shall be in units that can support 
the shortest temporal sampling of the video objects. It can 
also be the native time base for the object which is rendered. 
It therefore represents the finest granularity of temporal 
resolution that is required or can be supported. 

The VOP time increment can then be indicated by a finite 
length number greater or equal to the ratio of the global time 
base interval to the local time base resolution. FIG. 3A 
shows an example of the VOP time increment for I and 
P-video object planes and the reference to the moduio time 
base. An absolute time base is used. The VOP time incre- 
ment is reset each time the modulo time base is encountered. 
FIG. 3B shows anotiier example using I, P and B-video 
object planes. The operation is the same with the exception 
that the modulo time base is repeated in the B-video object 
planes. If the modulo time base is not repeated in the 
B-video object plane then ambiguities arise due to the 
difference in decoding and presentation order. This is 
described in detail below. 

Since the VOP time increment corresponds to the presen- 
tation time base, a potential problem wiLL arise when the 
encoding order is different from presentation order. This 
occurs with B-video object planes. Similar to MPEG-l and 
MPEG-2 B-pictures» B-video object planes are encoded 
after their reference I and P-video object planes even though 
they preceded the reference video objects in the presentation 
order. Since the VOP time increment is finite and is relative 
to the modulo time base, the VOP time increment is reset 
when the modulo time base is encoimtered. However, the 
order of encoding for the B-video object planes have been 
delayed. HG. 4 show the ambiguity that can arise. It is not 
possible to determine when the VOP time increment should 
be reset In fact when given a sequence of encoded events as 
in FIG. 4A, it is not possible to know which of the timing 
situations of FIGS. 4B, 4C and 4D it is trying to represent. 
This problem comes about because of the use of one modulo 
time base shared among all the different types of video 
object coupled with die difference between the encoding and 
presentation order. There is nothing that can be done to the 
encoding order since the reference information is required 
by the B-video object planes. It is also not desirable to have 
independent modtilo time base for the different prediction 
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type. Therefore the solution is to code the VOP time 
increment of the B-video object planes as a relative value to 
the preceding I or P-video object planes and that the moduio 
time base apphes only to the I and P-video object planes and 
5 not the B video object planes. This solution is depicted in 
FIG. 5. 

Next, VOP time offset is described. 

Besides the above the moduio time base are shared 
between all video object planes. This means that die syn- 
chronization of the different video objects will have a 
granularity equivalent to the modulo time base interval. This 
is unacceptable especially in the case where video object 
planes from different groups are combined to form a new 
group of video object planes. FIG. 6 shows an example of 
two different video object planes that were encoded with two 
different local time base that are offset from one another. 
Thus when the video object planes are multiplexed, the 
synchronization of the video objects planes are also offset. 
A finer granularity is achieved by allowing each individual 
video object plane to have a VOP time offset. This means 
that only this value is changed when the video object planes 
are manipulated and multiplexed. There is no need to change 
the VOP thne increment and yet it is possible to multiplex 
the different video object planes without coarse granularity 
timing differentials, FIG, 6 illustrates the use of tiiis time 
base offsets. 

The preferred embodiment of the current invention 
involves a method of encoding the time base for each of the 
individual video object plane bitstream, a method of multi- 
plexing different video object planes into a common time 
base, a method of de-multiplexing the multiplexed bitstream 
into the components and a method of recovering the time 
base from the component bitstream. 

35 Encoding die time base is described. 

The flow diagram of the embodiment of the time base 
encoding is shown ia FIG. 7. In the encoder the local time 
base is first initialized to the local start time in step 1. The 
process is passed to step 2 where the encoder determines the 

40 current value of the local time base. In step 3 the local time 
base obtained is compared to the previously encoded 
modulo time base to see if the interval has exceeded the 
modulo time base interval. If the interval has beentje^^^e-^ded 
the control is passed to step 4 where the required number of 

45 modulo time base is inserted into the bitstream. If the 
interval has not been exceeded then no special handling is 
necessary. The processing then proceeds to step 5 where the 
VOP tune increment is insened into die bitstream. The 
object is then encoded in step 6 and the inserted into die 

50 bitstream. The encoder then checks to determine if there are 
more objects to he encoded in step 7. If diere are more 
objects to be encoded the process resimies at step 2 where 
the local time base is obtained. If there are no more objects 
to be encoded tiien the process terminates. 

55 The following formula are used to determine the absolute 
and relative VOP time increments for I/P-video object 
planes and B-video object planes, respectively, 

60 taTBn'=^cTS&fc7rso(^^ 1, 2. 3, ... ) (i) 

65 where 

tcTBn is the encoder time base marked by the nth encoded 
moduio time base. 
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^GTB/ is the pre-determined modulo time base interval, 
^GTBo is the encoder time base start time, 
^AVT/ is the absolute VOP time increment for the I or P- video 
object planes, 

^ETBVp is the encoder time base at the start of the encoding 

of the I or P video object planes, 
t/fvrr is the relative VOP time increment for the B-video 

object planes, and 
Wss is the encoder time base at the start of the encoding of 

the B video object plane. 

Next, multiplexing more than one video object planes is 
described. 

When more than one video object planes are multiplexed 
together, the multiplexer examines the multiple video object 
plane bitstreams to determine the order of multiplexing as 
well as the synchronizadon. The operations involved is 
depicted in FIG. 8. The VOP dme offsets for each of the 
video object planes to be multiplexed are inserted into the 
bitstream in step 11. All the bitstreams of the video object 
planes to be multiplexed are then examined in step 12 to 
determine if all of the planes are at theu: respective modulo 
time base. If they are tiien the processing is passes to step 13 
where a common modulo time base is inserted into the 
multiplexed bitstream. The processing then proceeds to step 
14 ,where the next encoded video object is inserted into the 
multiplexed bitstream. In step 15 the bitstreams of the video 
object planes to be multiplexed are again examined to see if 
there are more video objects to be multiplexed. If there is, 
then the control is passed to step 12 again. Otiierwise the 
process ends. 

De-multiplexing the bitstream containing more than one 
video object planes is described. 

The de-muitiplexing of the bitstream containing multiple 
video object planes is depicted in FIG. 9. The process begins 
in step 21 where the VOP time offsets are decoded and 
passed to the decoders for synchronization. Then the mul- 
tiplexed bitstream is examined in step 22 to see if the 
modulo time base is found. If the modulo time base is found 
then the process is passed to step 23 where the modulo time 
base is inserted into all the video object bitstreams. The 
processing then continues with step 24 where the next video 
object is examined and inserted into the appropriate video 
object bitstream. Finally the multiplexed bitstream is exam- 
ined again to see if there is any more video object to 
de-multiplex. If tiiere is, then the process is passed to step 22 
again. Otherwise the process ends. 

Recovering the time base is described. 

The embodiment of the time base recovery is shown in 
FIG. 10. In recovering the local time base, the process 
begins in step 31 where the local time based is initialized 
taking into account of the VOP time offset decoded by the 
de-multiplexer. Then the process is passed to step 32 which 
checks the bitstream to determine if the modulo time base is 
decoded. If die modulo time base is decoded tiien the 
process is passed to step 33 where the local time base is 
incremented by the modulo time base increment. The pro- 
cess is then passed to step 37. If the modulo time base is not 
decoded then die process is passed to step 34 where the 
video object is examiae to determine if it is a B-video object 
or not If it is then the process is passed to step 35 where the 
B-video object decoding time base is calculated based on 
equation (6). The process is then passed to step 37. If the 
result of step 34 is not a B-video object then the process is 
passed to step 36 where the decoding time base is calculated 
based on equation (5). The process then proceeds to step 37. 
In step 37, the bitstream is examine to see if there is more 
video objects to decode. If there is then the process is passed 
to step 32 again. Otherwise the processing ends. 



The following formula are used to determine the presen- 
tation time stamp of the video objects: 
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where 

^GTBn is the decoding time base marked by the nth decoded 

modulo time base, 
^cTBi is ^e pre-determined modulo tune base interval, 
Wso is the decoding time base start time, 
^DT3i/F is the decoding time base at the start of the decoding 

of the I or P video object planes, 
t^vT-; is the decoded absolute VOP time increment for tiie I 

or P-video object planes, 
^DTBB is the decoding time base at the start of the decoding 
20 of the B video object plane, and 

t^vr/ is the decoded relative VOP time increment for the 
B-video object planes. 

Implementation of the Bitstream Encoder is described. 
FIG. 11 shows the block diagram for an implementation 
25 of the bitstream encoder for encoding the modulo time base 
and the VOP time increment For the purpose of this 
illustration the example in FIG. 3B is used. Since 
bi-directional prediction is used, the encoding order is 
different from die presentation order shown in FIG, 3B. The 
30 encoding order begins with the I- VOP followed by the 
P-VOP before the B-VOP. This is illustrated in the three 
paragraphs below. 

The process begins in the initializer, step 41 where the 
bitstream encoder starts by initializing the local time base 
35 register to the initial value of the time code. The same time 
code value is encoded into the bitstream. At the start of the 
encoding of the next I-VOP the time code comparator, step 
42, compares the presentation time of the I-VOP against the 
local time base register. The result is passed to the modulo 
40 time base encoder, step 43. The modido time base encoder 
will insert mto the bitstream the required number of "1" 
equivalent to the number of modulo time base increment that 
has daf.^ed. This is then followed by the symbol "O" to 
signal the end of the modulo time base code. The local time 
45 base register is updated to the current modiido time base. The 
processing then proceed to die VOP tune base increment 
encoder, step 44, where the remainder of the presentation 
time code of the l-VOP is encoded. 
The process then repeats with the next encoded video 
50 object plane which is the P-VOP. The time code comparator, 
step 42, compares the presentation time of the P-VOP 
against the local time base register. The result is passed to 
the modulo time base encoder, step 43. The modulo time 
base encoder will insert die required number of "1" equiva- 
55 lent to the number of modulo time base increment that has 
elapsed. This is then followed by the symbol "O" to signal 
die end of die modulo time base code. The B-VOP time base 
register is set to the value of the local time base register and 
the local time base register is updated to die current modulo 
60 time base. The processing &en proceeds to the VOP time 
base increment encoder, step 44, where die remainder of the 
presentation time code of the P-VOP is encoded. 

The process then repeats with die next encoded video 
object plane which is the B-VOP. Hie time code comparator, 
65 step 42, compares the presentation time of die B-VOP 
against die B-VOP time base register. The result is passed to 
die modulo tune base encoder, step 43. The modulo time 
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base encoder will insert the required number of "1" equiva- in the top row in FIG. 13, the compressed video data VOPs 
lent to the number of modulo time base increment that has are aligned in the display order, II, Bl, B2, PI, B3, P3 with 
elapsed. This is then followed by the symbol "O" to signal a GOP (group of pictures) header inserted at the beginning 
the end of the modulo time base code. Both the B-VOP time of group of VOPs. While being displayed, the local time, at 
base register and the local time base register are not changed 5 which the display is effected, is measured for each VOP 
after the processing of the B-VOR The processing then using the local time clock. For example, the first VOP 
proceed to the VOP time base inaement encoder, step 44, ai-VOP) is displayed at 1 hour 23 minute 45 second 350 
where the remainder of the presentation time code of the milHsecond (1:23:45:350) counted from the very beginning 
B-VOP is encoded of the video data; the second VOP (Bl-VOP) is displayed at 

The local time base register is reset at the next I-VOP lO 1:23:45:750; the third VOP (B2-V0P) is displayed at 
which marks the beginning of the next group of VOP. 1:23:46:150; and so on. 

Implementation of the Bitstream Decoder is described For encoding the VOPs, it is necessary to insert the 

FIG. 12 illustrates the block diagram for the implemen- display time data to each VOP. The insertion of the full time 
tation of the decoder for the modulo time base and VOP time data including hour, minute, second and milHsecond occu- 
increment to recover the presentation time stamp. As in the 15 pies a certain data space in the header portion of each VOP. 
implementation of die encoder tht example in HG. 3B is The aim of the present invention is to reduce such a data 
used. The decoding order is the same as the encoding order space and to simplify the time data to be inserted in each 
where the 1-VOP is decoded followed by the P-VOP before VOP. 

the B-VOP. This is explained in the paragraphs below. Each of the VOPs shown in the first row m HG. 13 is 

The processing begins in the initializer, step 51. where the 20 stored with,the display time data of milHsecond at the VOP 
local time base register is set to the value of the time code time increment area. Each of the VOPs in the first row is also 
decoded from the bitstream. The processing then proceeds stored temporarily with the display time data of hour, minute 
the modulo time base decoder, step 52, where the modulo and second. The GOP header is stored with the display data 
time base increment is decoded. The total number of modulo of hour, minute and second for the first VOP ai-VOP). 
time base increment decoded is given by the number of "i" 25 As shown in the second row in HG. 13, die VOPs are 
decoded before the symbol "0" Next the VOP time base . delayed by a predetermined time using a buffer (not shown), 
increment is decoded in the VOP time base increment, step When the VOPs are produced from the buffer, the order of 
53. In the time base calculator, step 54, the presentation time the VOPs are changed according to die bi-directional pre- 
of the I-VOP is recovered. The total decoded modulo time . diction rule such that the bi-directional VOP, i.e., B-VOP, 
base increment value is added to the local time base register. '30 should be positioned after die P-VOP to which the B-VOP 
The VOP time base increment is then added to the local time references. Thus, die VOPs are aHgned in die order of U, PI, 
base register to obtain die presentation time of the I-VOR Bl, B2, P2, B3. 

The process then goes to die video object decoder where the As shown in die dvird row in HO, 13, at time Tl. i.e., 
video object is decoded. when die GOP header is being encoded, die local time base 

For die P-VOP, die process repeats in die modulo time 35 register stores die hour, minute, second data as stored in die 
base decoder, step 52, where die modulo time base incre- * GOP header. In die example shown in HG. 13, die local time 
ment is decoded The total number of modulo time base - base register stores 1:23:45. Then, before time T2. die 
inaement decoded is given by die number of "1" decoded bitstream data for die GOP header is obtained widi die hour, 
before die symbol "0'\ Next die VOP time base increment , minute, second data being inserted as shown at die bottom 
is decoded in die VOP time base increment, step 53. In the 40 row in FIG. 13. 

time base calculator, step 54, die presentation time of die • Then, at time T2. die first VOP ai-VOP) is taken up. The 
P-VOP is recovered. The B-VOP modulo time base register time code comparator compares die time (hour, minute, 
is set to die value in die local ti^":-base register. The total second) stored in die local time base register widi die time 
decoded modulo time base increment value is added to die (hour, minute, second) temporanly stored in die first VOP 
local time base register. The VOP time base increment is 45 (Il-VOP). According to die example, die result of the 
dien added to die local time base register to obtain die . comparison is die same. Thus, die comparator produces **0" 
presentation time of die P-VOP. The process dien goes to die indicating diat die first VOP (Il-VOP) occurs in die same 
video object decoder where die video object is decoded. second as die second being held in die local time base 
For die B-VOP, die process repeats in die modulo time register. The result "0" as produced from die comparator is 
base decoder, step 52, where die modulo time base incre- 50 added to die first VOP (Il-VOP) at die modulo time base 
ment is decoded. The total number of modulo time base area. At die same dme, die hour, minute and second data 
increment decoded is given by die number of "1" decoded temporarily stored in die first VOP (Il-VOP) is eHminated 
before die symbol "O". Next die VOP time base increment Thus, before time T3, die bitstream data for die first VOP 
is decoded in die VOP time base increment, step 53, In die (Il-VOP) is obtained widi ^'O" inserted in die modulo time 
time base calculator, step 54, die presentation time of die 55 base area and "350" inserted m die VOP time mcrement 
B-VOP is recovered. The total decoded modulo time base area. 

inaement value and die VOP time base inaement is added Then, at time T3, die second VOP (Pl-VOP) is taken up. 
to the B-VOP time base register to obtain die presentation The time code comparator compares die time (hour, ramute, 
time of die B-VOP. Bodi die B-VOP time base register and second) stored in die local time base register widi die time 
die local time base register are left unchanged The process 60 (hour, minute, second) temporarily stored m die second V<3P 
dien goes to die video object decoder where die video object (Pl-VOP). According to die example, die result of the 
is decoded comparison is such diat die time temporarily stored in the 

The local time base register is reset at die next I-VOP second VOP (Pl-VOP) is one second greater dian die time 
which marks die beginning of die next group of VOP. stored m local time base register. Thus, die comparator 

Specific example is described, 65 produces "10" indicating diat die second VOP (Pl-VOP) 

Referring to FIG. 13, an example of die steps for encoding occurs in die next one second to die second being held in die 
die compressed data into bitstream data is shown. As shown local time base register. If die second VOP (Pl-VOP) occurs 
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in the next— next second to the second being held m the After time T6, the B-VOP time base register is incre- 
local time base register, the comparator will produce "liO'\ mented to the time equal to the time carried in the local time 
After time T3> the B-VOP time base register is set to the base register immediately before time T6. In the example^ 
time equal to the time carried m the local time base register the B-VOP time base register is incremented to 1:23:46, 
immediately before time T3. In the example, the B-VOP 5 Also, after time T6, the local time base register is incre- 
time base register is set to 1 :23:45. Also, after time T3, the mented to the time equal to the time temporarily stored in the 
local time base register is mcremented to the time equal to f^f^ vOP (P2-V0P). Thus, in the example, die local time 
the time temporarily stored in the second VOP (Pl-VOP). register is incremented to 1:23:47. 

Thus, in the example, the local time base register is incre- ^^^^^ » ^ produced from the comparator is added 

mented to 1:23:46. ^ ^ 10 to the fifdi VOP (P2-V0P) at the modulo time base area. At 

The result "10" as produced from the comparator is added ^^^^ ^^^^ ^^^^^^ temporarily 

to the second VOP (Pl-VOP) at the modulo time base area. ^^^^^ (p2.vOP) is eliminated. Tbus, before 

At the same time the ^^^^^^^^^^^^^ time T4, the bitstream data for the fifth VOP (P2-V0P) is 

maly stored m the second VOP (P™P) ^ obtainedwith "10" inserted m the modulo time base area and 

Thus, before tune T4, the bitstream data for the second VOP ^ . ^ , . ^ , 
(Pl-VOP) is obtained with "10" inserted in the modulo time 15 "350' mserted m the VOP time mcrement area 
base area and ^'55^ inserted in the VOP time increment . Thereafter, a smidar operauon is caxn^^ out for formmg 

bitstream data for the followmg VOPs. 
Then at time T4, the third VOP (Bl-VOP) is taken up. For decodmg the bitstream data, an operation opposite to 
The time code comparator compares the time (hour, minute, the above is carried out. First, the time (hour, mmute, 
second) stored in the B-VOP time base register with the time 20 second) carried in the GOP header is read. The read time is 
(hour, minute, second) temporarily stored in the third VOP stored in the local time base register. 
(Bl-VOP), According to the example, the result of the Upon receipt of the 1 type or P type VOP, Le., other then 
comparison is the same. Thus, the comparator produces "0" the B type VOP, the data stored in the modulo time base area 
mdicating that the third VOP (Bl-VOP) occurs in the same is read. If the read data is "0'\ i.e., without any is in front of 
second as the second being held in the B-VOP time base 25 0, no change is made in die local time base register, and also 
register. The result "0" as produced from die comparator is . no change is made in the B-VOP time base register. If the 
added to die third VOP (Bl-VOP) at the modulo time base ' read data is "10", the time stored in die local time base 
area. At die same time, die hour, minute and second data register is incremented by one second. If die read data is 
temporarily stored m±ediird VOP (Il-VOP) is elhninated. ,"110", die time stored in die local time base register is 
Thus, before time T5, die bitstream data for die diird VOP 30' incremented by two seconds. In diis manner, die number of 
(Bl-VOP) is obtained widi "0" mserted in die modulo time seconds which should be incremented is determined by die 
base area and "750" mserted in die VOP time increment number of is inserted in front of 0. Also, when die read data 

is "10" or "110", the B-VOP time base register, which is a 
Then, at time T5, die fourth VOP (B2-V0P) is taken up. memory, copies die time die local time base register had 
The time code comparator compares the time (hour, minute, 35 carried just before die most recent increment Then, die time 
second) stored in die B-VOP time base register widi die time * (hour, minute, second) carried in die local time base register 
(hour, minute, second) temporarily stored m die fourth VOP is combined widi die time (millisecond) carried m die VOP 
(B2-V0P). According to die example, die result of die - time increment area to establish a specific time at which die 
comparison is such diat die time temporarOy stored in die I type or P type VOP should occur, 
fourdi VOP (B2-V0P) is one second greater dian die time 40* Upon receipt of die B type VOP, die data stored m die 
stored in B-VOP time base register. Thus, die comparator • modulo time base area is read If die read data is "0", die 
produces "10" indicating diat die fourth VOP (B2-V0P) time (hour, minute, second) carried in die B-VOP VOP time 
occurs in die next one second to die second being held m die base register is combined widi die time (miUisec^^.d) carried 
B-VOP time base register. ^ time increment area to establish a specific time 

E>uiing die processing of die B type VOP, neidier die local 45 at which die B type VOP should occur. If die read data is 
time base register nor die B-VOP time base register is "10", die time (hour, minute, second) carried in die B-VOP 
incremented regardless of whatever result die comparator time base register is added one second, and dien die added 
produces. result time is combined with die time (millisecond) carried 

The result "10" as produced from die comparator is added in die VOP time increment area to establish a specific time 
to die fourth VOP (B2-V0P) at die modulo time base area, 50 at which die B type VOP should occur. If die read data is 
At die same time, die hour, minute and second data tempo- "110", die time (hour, minute, second) carried in die B-VOP 
rariiy stored m die fourdi VOP (B2-V0P) is eliminated. time base register is added two seconds, and dien die added 
Thus, before time T6, the bitstream data for die fourth VOP result time is combined widi die time (millisecond) carried 
(B2-VOP) is obtained widi "10" inserted in die modulo time in die VOP time increment area to establish a specific time 
base area and "150" inserted in die VOP time increment 55 at which die B type VOP should occur. 

The effects of diis invention is to enable video object 
Then, at time T6, die fifdi VOP (P2-V0P) is taken up. The planes encoded by different encoders to be multiplexed. It 
time code comparator compares die time (hour, minute, wiU also facilitate object based bitstream manipulation of 
second) stored in die local time base register widi die time compressed data diat come from different sources to create 
(hour, minute, second) temporarily stored in die fifdi VOP 60 new bitstreams. It provides a mediod of synchronization for 
(P2-V0P). According to die example, die result of die audio visual objects. 

comparison is such diat die time temporarily stored in die The invention bemg dius described, it wiU be obvious diat 
fifdi VOP (P2-V0P) is one second greater dian die time die same may be varied in many ways. Such variations are 
stored in local time base register. Thus, the comparator not to be regarded as a departure from die spirit and scope 
produces "10" indicating diat die fifdi VOP (P2-V0P) 65 of die invention, and all such modifications as would be 
occurs in die next one second to die second being held in die obvious to one skilled in die art are intended to be included 
local time base register. widiin die scope of die following claims. 
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/We claim: 

■ 1. A method of encoding a local time base of [an audio] a 
visual sequence in the compressed data comprising the steps 
of: 

obtaining instances of the [audio] visual sequence by tern- ^ 
porai sampling; 

determining the local time base of the said instances to be 
encoded into compressed data; 

encoding the said local time base in two parts, comprising 
of a modulo time base that marks die occurrence of a 
set of time reference spaced evenly at a specific interval 
on the local time base and a time base increment 
relative to the said evenly spaced time reference; 

inserting the modulo time base into the compressed data J5 
whenever the specific interval have elapsed; and 

inserting the time base increment within the compressed 
data of the said instances of the [audio]visual sequence. 

2. A method of claim 1, wherein a plurality of compressed 
bitstreams containing local time base information encoded 20 
therein are multiplexed and de -multiplexed, for 
de-multipiexing further comprising the steps of: 

decoding the time base offset of the individual com- 
pressed bitstream; 

examining the multiplexed bitstream for the next com- " 
pressed instance and placmg the said instance into the 
appropriate compressed bitstream mitil a modulo time 
base is encountered m the multiplexed bitstream; 

inserting a modulo time base to each of the compressed 
bitstreams; and 

repeating the last two steps until the multiplexed bitstream 
is exhausted. 

3. A method of decoding a local time base of [an audio] a 
visual sequence from the time base of the compressed data 35 
encoded according to claim 1 comprising the steps of: 

mitializing the reference time base taking into account the 

time base offset; 
incrementing the reference time base by a specific interval 

for each modulo time base decoded; 
decoding the time base increment of the compressed 

instance; and 

determining the decoding time base of the said instance 
by adding the said decoded time base increment value 
to the reference time base. 

4. A method of encoding the time base according to claim 
1 where the local time base, time base increment and time 
base offsets are in units of milliseconds and the specific 
interval has a dmration of 1000 milliseconds, 

5. A method of claim 1. wherein a plurality of compressed 
bitstreams containing local time base information encoded 
therein are multiplexed, for multiplexing further comprising 
the steps of: 

synchronizing the local time base of the individual com- 55 
pressed bitstream to a conmion time base by encoding 
and inserting the time base offset in the multiplexed 
bitstream; 

examining each of the compressed bitstream for the next 
compressed instance to be placed into the multiplexed 60 
bitstream until all the compressed bitstreams have 
arrived at the modulo time base; 
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inserting a common modulo time base into the multiplex 
bitstream and skipping the modulo time base of the 
compressed bitstreams; and 
repeating the last two steps until all compressed bit- 
streams are exhausted. 
6, A method of multiplexing a plurality of compressed 
bitstreams as in claim 5 where the individual compressed 
bi^treams are themselves multiplexed bitstreams. 
J7. A method of encoding a local time base of [an audio] a 
visual sequence in the compressed data comprising the steps 
of: 

obtaining instances of the [audio] visual sequence by tem- 
poral sampling; 

determining the local time base of the said instances to be 
encoded into compressed data; 

encoding the said instances in one of two methods com- 
prising of a first method of compression without ref- 
erence to any future instances and a second method of 
compression which refers to the future reconstructed 
instance; 

encoding the said local time base in two parts, comprising 
of a modulo time base that marks the occurrence of a 
set of time reference spaced evenly at a specific interval 
on the local time base and a time base increment; 

encoding the time base increment for the instance com- 
pressed using the first compression method as an abso- 
lute value relative to the said evenly spaced time 
reference; 

encoding the time base increment for the instance com- 
pressed using the second compression method as a 
relative value to the local time base of the previously 
compressed instance using the said first method; 

inserting the modulo time base into the compressed data 
whenever the specific interval have elapsed; and 

inserting die time base increment within the compressed 
data of the said instances of the [audio] visual sequence. 

8. A method of decoding a local time base of [an audio] a 
visual sequence from the time base of the compressed data 
encoded according to claim 7 comprising the steps of 

initializing the reference time base taking into account of 
the time base offset; 

incf^"- -entLng the reference time base by a specific interval 
for each modulo time base decoded; 

decoding the time base increment of die compressed 
instance; and 

determining the time base increment to be one of two 
types, absolute or relative, depending on the first or 
second compression method used in the encoding of the 
instance, respectively; 

determining the decoding time base of the said instance 
by adding the said decoded time base increment value 
to the reference time base if the time base is of the first 
type; and 

determining the decoding time base of the said instance 
by adding the said decoded time base increment value 
to the decoding time base of the previous instance 
encoded using the first compression method, if the time 
base increment is of the second type. 

***** 



